feat: BackendCpp: C++ source-generation backend for OSL shader groups#2130
Open
lgritz wants to merge 1 commit into
Open
feat: BackendCpp: C++ source-generation backend for OSL shader groups#2130lgritz wants to merge 1 commit into
lgritz wants to merge 1 commit into
Conversation
Add a BackendCpp execution path that generates human-readable, compilable
C++ from the post-optimized shader graph, as an alternative to the LLVM
JIT.
The direct benefit is for debugging and understanding post-optimized
shader by looking at its C++ equivalent rather than huge a pile of oso.
Possible secondary/future benefits (with a little more work):
- Speed, if DSO loading is faster than optimize/JIT.
- Pre-compiled shader networks for use in a renderer that needs a
fixed set of shaders and no runtime JIT.
- Distributing proprietary shaders as binaries.
But the REAL purpose is as a stepping stone to translating to other
languages like Metal or GLSL. I envision the mechanism to simply be to
subclass BackendCpp, inheriting all the parts that are common to all
C++-like languages, and using virtual methods as the customization
points for language-specific differences. (I don't claim that these
customization points are correct now; I expect significant refactoring
needs to be apparent when we add the next language.)
Anyway, so as for what's here:
Pipeline, controlled by the escalating `debug_output_cpp` ShadingSystem
attribute (and the `OSL_DEBUG_OUTPUT_CPP` env var):
1 - generate a self-contained `group-cpp-<name>.cpp`
2 - also shell out to compile it to a DSO
3 - also load the DSO, verify its ABI, and execute it instead of the JIT
(running BackendLLVM's layout pass only, skipping full JIT codegen)
Generated output: a typed `GroupData` struct mirroring the BackendLLVM
layout (layer-run flags, userdata, connected/output params), one
`void layer_N(ShaderGlobals*, GroupData*, ...)` function per active layer,
and a `RunLLVMGroupFunc`-compatible group entry. Every file includes the
new internal `osl_cpp_runtime.h` (ABI version + `extern "C"` osl_* decls)
and exports `osl_cpp_abi_version()`, checked at load. The `lang_*` virtual
interface keeps the language tokens overridable for future backends.
Op coverage is complete: arithmetic/comparison/math emit direct C++;
runtime-service ops (noise, texture, closures, transforms, pointcloud,
getattribute/getmatrix/gettextureinfo, splines, regex, messages, ...) call
osl_* functions. Control flow, arrays/components, structs (incl.
color2/4, vector2/4), and Dual2 derivatives (scalar and triple) are all
generated and verified against the JIT.
Testing is opt-out and automatic: the `OSL_TEST_CPP_BACKEND` CMake option
(ON in the linux-vfx2026 and macos26-arm CI variants) builds a per-group
DSO and runs every testshade/testrender test through the C++ path at both
opt levels. The full eligible suite passes, matching JIT output (with the
documented opt-outs: --entry/layers-entry, the backend-cpp fixture itself,
OptiX/GPU, and batched-regression harnesses).
Cross-platform notes:
- osl_* shadeops are exported from liboslexec for generated DSOs to
resolve. On Linux this required listing osl_* under `global:` in
hidesymbols.map (the version script otherwise localized them out of
the dynamic symbol table); they are marked INTERNAL/UNSTABLE.
- OSL_CPP_ABI_VERSION folds in the OSL major/minor version so minor
releases are link-incompatible automatically (the DSOs are ephemeral;
the check only guards against misuse).
- MSVC-clean (OSL::popcount instead of a GCC/Clang builtin).
- Array copy of mismatched lengths copies min(dst,src) elements,
matching BackendLLVM and avoiding an out-of-bounds source read.
Spec, plan, research, data-model, and tasks under
docs/dev/specs/002-backend-cpp/.
Assisted-by: Claude Code / claude-sonnet-4-6 and claude-opus-4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a BackendCpp execution path that generates human-readable, compilable C++ from the post-optimized shader graph, as an alternative to the LLVM JIT.
The direct benefit is for debugging and understanding post-optimized shader by looking at its C++ equivalent rather than huge a pile of oso.
Possible secondary/future benefits (with a little more work):
But the REAL purpose is as a stepping stone to translating to other languages like Metal or GLSL. I envision the mechanism to simply be to subclass BackendCpp, inheriting all the parts that are common to all C++-like languages, and using virtual methods as the customization points for language-specific differences. (I don't claim that these customization points are correct now; I expect significant refactoring needs to be apparent when we add the next language.)
Anyway, so as for what's here:
Pipeline, controlled by the escalating
debug_output_cppShadingSystem attribute (and theOSL_DEBUG_OUTPUT_CPPenv var):1 - generate a self-contained
group-cpp-<name>.cpp2 - also shell out to compile it to a DSO
3 - also load the DSO, verify its ABI, and execute it instead of the JIT
(running BackendLLVM's layout pass only, skipping full JIT codegen)
Generated output: a typed
GroupDatastruct mirroring the BackendLLVM layout (layer-run flags, userdata, connected/output params), onevoid layer_N(ShaderGlobals*, GroupData*, ...)function per active layer, and aRunLLVMGroupFunc-compatible group entry. Every file includes the new internalosl_cpp_runtime.h(ABI version +extern "C"osl_* decls) and exportsosl_cpp_abi_version(), checked at load. Thelang_*virtual interface keeps the language tokens overridable for future backends.Op coverage is complete: arithmetic/comparison/math emit direct C++; runtime-service ops (noise, texture, closures, transforms, pointcloud, getattribute/getmatrix/gettextureinfo, splines, regex, messages, ...) call osl_* functions. Control flow, arrays/components, structs (incl. color2/4, vector2/4), and Dual2 derivatives (scalar and triple) are all generated and verified against the JIT.
Testing is opt-out and automatic: the
OSL_TEST_CPP_BACKENDCMake option (ON in the linux-vfx2026 and macos26-arm CI variants) builds a per-group DSO and runs every testshade/testrender test through the C++ path at both opt levels. The full eligible suite passes, matching JIT output (with the documented opt-outs: --entry/layers-entry, the backend-cpp fixture itself, OptiX/GPU, and batched-regression harnesses).Cross-platform notes:
global:in hidesymbols.map (the version script otherwise localized them out of the dynamic symbol table); they are marked INTERNAL/UNSTABLE.Spec, plan, research, data-model, and tasks under docs/dev/specs/002-backend-cpp/.
Assisted-by: Claude Code / claude-sonnet-4-6 and claude-opus-4.8